Exploring Medalla data

Initial exploration.

Table of Contents

• Data sources
  • Lighthouse block export
  • Weald dump
  • Computed datasets
• Performance of duties
  • Attester duties
  • Proposer duties
• Correctness of attestations
  • Target checkpoint
  • Head of the chain
• Validator performance
  • Uptime-rewards curve I: Included attestations
  • Uptime-rewards curve II: Inclusion delay
• Aggregate attestations
  • How many individual attestations are contained in aggregates?
  • In how many aggregate attestations is a single attestation included?
  • How many redundant aggregate attestations are there?
  • How many times did a block include the exact same aggregate attestation more than once?
  • How many times were clashing attestations included in blocks?

---

In this notebook we explore data from the Medalla testnet. We are looking at the 388001 first slots.

You can find the source of this notebook over at the RIG repository. The code to generate the datasets is also available.

Data sources

Lighthouse block export

We use a fork of Lakshman Sankar’s Lighthouse block exporter to export attestations and blocks from the finalised chain until slot 388000.

We present the main datasets below:

all_ats

Each row in this dataset corresponds to an aggregate attestation included in a block.

exploded_ats

We cast the dataset above into a long format, such that each row corresponds to an individual attestation included in a block. Note that when this individual attestation is included multiple times over multiple aggregates, it appears multiple times in the dataset.

individual_ats

exploded_ats is the “disaggregated” version of the aggregate attestations. To check for validator performance, we often don’t need to check for every inclusion of their individual attestations. individual_ats contains these unique, individual attestations, tagged with some extra data such as their earliest inclusion and whether they attested correctly for the target checkpoint and the head.

all_dps

Validators are allowed to deposit more ETH into their eth2 accounts. Later on, we compute the reward obtained by a validator by comparing its initial balance with its current balance. We must then deduct deposits that were made since genesis.

The validator index is not included in the Deposit object recorded on the beacon chain. We use data from the Weald dump, presented below, to recover the validator index from its public key.

Weald dump

Jim McDonald, from Attestant, kindly provided a treasure trove of data on the #medalla-data-challenge channel of the EthStaker Discord server. The two previous datasets could have legitimately been mined from Jim’s data, but we like to get our hands dirty.

all_cms

Not too dirty though: obtaining the past record of committees (which validators are supposed to attend when) is much more computationally intensive, since it requires access to past states. Yet given that we have 12125 epochs in our dataset and a maximum of 73258 validators, a dataset compiling all committee assignments would have 888253250 rows, which is too much. When we need committee information, we’ll pull it from the database and record intermediary datasets instead.

Note that you can obtain past states from the Lighthouse block exporter too, but still need to compute the committees from them.

val_balances

This dataset gives us validator state balances at the beginning of each epoch. Note that the state balance (balance), the true ETH amount a validator deposited, is different from the effective balance (effective_balance), which measures the principal on which validators receive an interest.

Rewards are not given out at the end of genesis epoch, so the balance would only change for rows where epoch >= 2.

Computed datasets

To ease the computational demands of this notebook, we record two datasets from which much of the analysis can be derived.

stats_per_val

For each validator, we compute a bunch of statistics, including:

• included_ats: The number of times their attestations were included
• first_att/last_att: The attesting slot of their earliest and latest attestation (used by pintail to build validator types)
• correct_targets/correct_heads: How many times they correctly attested for the target checkpoint or the head
• avg_delay: Their average inclusion delay

stats_per_slot

We also record summary statistics for each slot. At 388000 slots in our dataset, this remains manageable to query. We have the following fields:

• included_ats: How many attestations were received for the slot.
• expected_ats: How many attestations were expected for the slot.
• correct_targets/correct_heads: The number of correct target/head attestations for that slot.

Performance of duties

Attester duties

We compare the number of included attestations with the number of expected attestations.

Clearly something went very wrong circa epoch 2,500. This is now known as the roughtime incident, an issue affecting the major validator client, Prysm. It took time for the network to recover, in the process demonstrating how the quadratic inactivity leak mechanism works. Client diversity FTW!

Proposer duties

How many blocks are there in the canonical chain?

Again, the same trough during the roughtime incident.

Correctness of attestations

Target checkpoint

Attestations vouch for some target checkpoint to justify. We can check whether they vouched for the correct one by comparing their target_block_root with the latest known block root as of the start of the attestation epoch (that’s a mouthful). How many individual attestations correctly attest for the target?

How does the correctness evolve over time?

Head of the chain

Attestations must also vote for the correct head of the chain, as returned by the [GHOST fork choice rule]. To check for correctness, one looks at the latest block known as of the attestation slot. Possibly, this block was proposed for the same slot as the attestation att_slot. When the beacon_block_root attribute of the attestation and the latest block root match, the head is correct!

How does the correctness evolve over time?

Validator performance

Validators are rewarded for their performance, and penalised for failing to complete their tasks. We start with a crude measure of performance: the number of included attestations. It is a crude measure since (a) we do not discount the timeliness of the validator, measured by the inclusion delay and (b) we do not check that the attestation’s attributes are correct (with the exception of the source attribute, since an incorrect source cannot possibly be included on-chain).

Uptime-rewards curve I: Included attestations

We compare the percentage of included attestations with the (possibly negative) reward obtained by the validator.

Who are the validators getting a negative return? We plot the same, showing how long a validator has been in service.

We only show return rates within one standard deviations of the mean. Very low reward rates are incurred by slashed validators.

Recently activated validators have a much more balanced uptime-reward curve, with the higher performers getting positive returns. Meanwhile, validators who were active since the beginning tend to have smaller returns. This can be due to validator fatigue (validating for a while, then turning off the rig), but a fair number of early validators have high attestation performance yet low return. The roughtime incident is likely to blame here. Let’s focus on these early validators.

Inactivity leaks push the uptime-rewards curve downwards. At best, validators can preserve their current balance if they validate optimally, with inclusion delay at 1 always. Most likely, active validators lose a small amount of their balance due to delay or attestation errors, while inactive validators leak much more.

Uptime-rewards curve II: Inclusion delay

We turn our attention to the inclusion delay. Validators are rewarded for attesting timely, with higher rewards the earlier they are included in a block. We explode aggregates contained in the blocks to trace the earliest included attestation of each validator in an epoch.

Note that the y axis is given on a logarithmic scale. A high number of attestations have a low inclusion delay, which is good! Since attestations cannot be included more than 32 slots from their attesting slot, the distribution above is naturally capped at 32.

How is the inclusion delay correlated with the rewards? We look at validators with at least 70% of included attestations and activated after the roughtime incident to find out.

The plot looks rather homogeneous…

Aggregate attestations

eth2 is built to scale to tens of thousands of validators. This introduces overhead from message passing (and inclusion) when these validators are asked to vote on the canonical chain. To alleviate the beacon chain, votes (a.k.a. attestations) can be aggregated.

In particular, an attestation contains four attributes:

• The slot it is attesting for.
• Its vote for the head of the beacon chain, given by the fork choice rule.
• Its vote for the source, i.e., the last justified checkpoint in its view.
• Its vote for the target, i.e., the checkpoint to be justified in its view.

Since we expect validators to broadly agree in times of low latency, we also expect a lot of redundant attestations. We can aggregate such a set of attestations \(I\) into a single, aggregate, attestation.

For each slot \(s\), a committee of validator \(C(s)\) is determined who is expected to attest for \(s\). Assume that two aggregate attestations were formed from validators attesting for \(s\), one aggregate of validators in set \(I \subseteq C(s)\) and the other with validators in set \(J \subseteq C(s)\). We have two cases:

• When the intersection of \(I\) and \(J\) is non-empty, we cannot aggregate the two aggregates further.
• When the intersection of \(I\) and \(J\) is empty, the two aggregates can themselves be aggregated, into one containing attestations from validator set \(I \cup J\).

In the following, we look at redundant, clashing and individual attestations.

How many individual attestations are contained in aggregates?

A fairly high number of aggregate attestations included in a block are actually individual attestations. Nonetheless, a significant number of aggregates tally up between 50 and 100 attestations.

We can plot the same, weighing by the size of the validator set in the aggregate, to count how many individual attestations each size of aggregates included.

Overall, we can plot the Lorenz curve of aggregate attestations. This allows us to find out the share of attestations held by the 20% largest aggregates.

The answer is 47%.

How much savings did aggregates provide?

We compare how many individual attestations exist to how many aggregates were included in blocks.

We have 23.84 times more individual attestations than aggregates, meaning that if we were not aggregating, we would have 23.84 as much data on-chain.

In how many aggregate attestations is a single attestation included?

We look at all individual attestations in our dataset, i.e., individual, unaggregated votes, and measure how many times they were included in an aggregate.

Most attestations were included in an aggregate once only

How many redundant aggregate attestations are there?

We call redundant identical aggregate attestations (same four attributes and same set of validator indices) which are included in more than one block. It can happen when a block producer does not see that an aggregate was previously included (e.g., because of latency), or simply when the block producer doesn’t pay attention and greedily adds as many aggregates as they know about.

The mode is 1, which is also the optimal case. A redundant aggregate does not have much purpose apart from bloating the chain.

How many times did a block include the exact same aggregate attestation more than once?

We could call these strongly redundant, as this is pure waste.

We see that 229 times, identical aggregates were included twice in a block.

How many times were clashing attestations included in blocks?

We define clashing attestations as two aggregate attestations included in the same block, with identical attributes (same attesting slot, beacon chain head, source block and target block). We can further define the following two notions, assuming the two aggregate attestations include attestations of validator sets \(I\) and \(J\) respectively:

• Weakly clashing: the two aggregates have different validator indices, \(I \neq J\).
• Strongly clashing: \(I \neq J\) and \(I \cap J \neq \emptyset\). The two aggregate attestations were incompatible, so could not be aggregated further.

We obtain how many times an attestation weakly clashes with itself, i.e., is included multiple times in a single block with different validator sets. For instance, if an aggregate attestation is included in the same block three times with a different set of validator indices each time, we record that this aggregate is weakly clashing three times with itself. We give below the histogram of this measure.

From the plot above, we observe that some aggregates were included over 40 times in the same block, all with different sets of validator indices. Still, most aggregates were included once or a few times.

Finding weakly clashing attestations that are not strongly clashing (i.e., which could have been aggregated further) is left for future work as it is more computationally intensive. In particular, for a set of aggregates identical up to their validator indices, one must find which have an empty overlap.

Note that optimally aggregating a set of aggregates is NP-complete! Here is a reduction of the optimal aggregation problem to the graph colouring. Set aggregate attestations as vertices in a graph, with an edge drawn between two vertices if the validator sets of the two aggregates have a non-empty overlap. In the graph colouring, we look for the minimum number of colours necessary to assign a colour to each vertex such that two connected vertices do not have the same colour. All vertices who share the same colour have an empty overlap, and thus can be combined into an aggregate. The minimum number of colours necessary to colour the graph tells us how few aggregates were necessary to combine a given set of aggregates further.